What happened?

The space shuttle Challenger was ready to launch on the morning of Tuesday, January 28, 1986.

Figure 1. Prepared to launch

This was a symbolic flight.

Figure 2. The crew

The night before the launch there was a three hour teleconference between Morton Thoikol, Marshall Space Flight Center, and Kennedy Space Center. The subject of the discussion was the sensitivity of O-rings, which seal the joints in booster rockets, to lower temperatures.

Figure 3. O-rings seal joints on booster rockets

The discussion considered the data below.

Figure 4. O-ring failure as a function of temperature

Figure 4. O-ring failure as a function of temperature

  • Upon inspection of this plot, most of the participants concluded that, since the plot is “U” shaped, there was no evidence that temperature affected O-ring failure.

  • Lesson: the absence of evidence \(\neq\) evidence of absence.

  • In contrast, at least one participant, Roger Boisjoly, stated that “temperature was indeed a discriminator”.

Regardless, the Challenger was launched the following morning.

Figure 5. Challenger launch (NASA)

73 seconds into the flight, the shuttle exploded.

Figure 6. The O-rings failed

What should have happened?

To understand the message from the committee, let’s look at the data:

Flight Temp Pressure O.ring Number
1 66 50 0 6
2 70 50 1 6
3 69 50 0 6
4 68 50 0 6
5 67 50 0 6
6 72 50 0 6
7 73 100 0 6
8 70 100 0 6
9 57 200 1 6
10 63 200 1 6
11 70 200 1 6
12 78 200 0 6
13 67 200 0 6
14 53 200 2 6
15 67 200 0 6
16 75 200 0 6
17 70 200 0 6
18 81 200 0 6
19 76 200 0 6
20 79 200 0 6
21 75 200 2 6
22 76 200 0 6
23 58 200 1 6
  • The number of O-rings that failed during 23 total flights was recorded with the temperature and pressure of the joints at launch.

Notice that some observations, those where 0 O-ring failures occured, were omitted.

Figure 7. O-ring failure as a function of temperature

Figure 7. O-ring failure as a function of temperature

  • Lessons:
    • ommission of outliers can be dangerous
    • outlier detection can be conditional on the inclusion/omission of other observations

O-ring failure might also be sensitive to pressure.

Figure 8. Number of O-ring failures as a function of temperature and pressure

  • Lesson: sometimes its valuable to consider several explanatory variables that may contribute to the response.

With the full dataset, another relationship(s) is apparent. To test the influence of temperature and pressure on O-ring failure, we have a couple options.

  • One option is to model the number of O-ring failures as a function of temperature and pressure.
    • Are there any problems with this approach?
  • Another option is to model O-ring failure as a binary function of temperature and pressure.
    • Are there any problems with this approach?
Dependent variable:
binomial binary
No.of O-ring failures O-ring failure
Temp -0.098** -0.229**
(0.045) (0.110)
Pressure 0.008 0.010
(0.008) (0.009)
Constant 2.520 13.292*
(3.487) (7.664)
Observations 23 23
Log Likelihood -15.053 -9.391
Akaike Inf. Crit. 36.106 24.782
Note: p<0.1; p<0.05; p<0.01

Consider the binomial model below:

\[ logit\left(\hat{\pi}\right)= 2.520 - 0.098\text{ temperature} + 0.008\text{ pressure} \] - And the inverse odds ratio:

##               Estimate Std. Error   z value  Pr(>|z|)
## (Intercept) 0.08044395 0.03059912 0.4853986 0.6251197
## Temp        1.10329014 0.95610229 8.9325907 0.9718580
## Pressure    0.99155187 0.99235230 0.3311752 0.7640570
  • For the binomial logistic regression model with temperature and pressure included, we see that a unit decrease in temperature changes the odds of failure by 1.1 times.

Now consider the binary model below:

\[ logit\left(\hat{\pi}\right)= 13.292 - 0.229\text{ temperature} + 0.010\text{ pressure} \] - And the inverse odds ratio:

##                 Estimate   Std. Error   z value  Pr(>|z|)
## (Intercept) 1.687336e-06 0.0004694408 0.1765067 0.9204913
## Temp        1.256928e+00 0.8958446419 7.9968563 0.9630857
## Pressure    9.896537e-01 0.9910609925 0.3140357 0.7813261

  • For the binary logistic regression model with temperature and pressure included, we see that a unit decrease in temperature changes the odds of failure by 1.25 times.

  • Lesson: temperature matters!

Does pressure actually matter?

  • To determine if both explanatory variables matter, we test \(H_0: \beta = 0\) vs \(H_A:\beta \neq 0\), for both temperature and pressure, for the binary model:
## Analysis of Deviance Table (Type II tests)
## 
## Response: Failure
##          LR Chisq Df Pr(>Chisq)   
## Temp       7.7542  1   0.005359 **
## Pressure   1.5331  1   0.215648   
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
  • Lesson: temperature matters!
  • Should we really remove pressure from the model? What are the advantages and disadvantages?

Temperature matters but… how confident are we that cooler temperatures really increase the odds of O-ring failure?

Figure 9. O-ring failure as a function of temperature

Figure 9. O-ring failure as a function of temperature

  • Why is the confidence band wider at lower temperatures than at higher temperatures?

How confident are we that O-rings would fail at a cold temperature, say 29°F, and a warm temperature, say 72°F?

Figure 10. 90% confidence interval estimated with a parametric bootstrap approach

Figure 10. 90% confidence interval estimated with a parametric bootstrap approach

What did we miss?

Resources: